Temperature compensation for structured light depth imaging system

ABSTRACT

Disclosed are an apparatus and a method of compensating temperature shifts of a structured light pattern for a depth imaging system. In some embodiments, a depth imaging device includes a light source, an imaging sensor and a processor. The light source emits light corresponding to a pattern. A temperature drift of the light source can cause a shift of the pattern. The imaging sensor receives the light reflected by environment in front of the depth imaging device and generates a depth map including a plurality of pixel values corresponding to depths of the environment relative to the depth imaging device. The processor estimates the shift of the pattern based on a polynomial model depending on the temperature drift of the light source. The processor further adjusts the depth map based on the shift of the pattern.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of U.S. Non-Provisional applicationSer. No. 15/251,966, entitled “Temperature Compensation for StructuredLight Depth Imaging System” and filed on Aug. 30, 2016, which isexpressly incorporated by reference herein in its entirety.

BACKGROUND

Depth sensing technology can be used to determine a person's location inrelation to nearby objects or to generate an image of a person'simmediate environment in three dimensions (3D). One application in whichdepth sensing technology may be used is in head-mounted display (HMD)devices and other types of near-eye display (NED) devices. Depth sensingtechnology can employ a stereo vision, time-of-flight (ToF) orstructured light depth camera. Such a device can create a map ofphysical surfaces in the user's environment (called a depth image ordepth map) and, if desired, to render a 3D image of the user'senvironment.

A depth sensing system (also referred to as depth imaging system) caninclude a light source for providing structured light. Structured lightis a process of projecting a known pattern of light onto a scene. Thelight is reflected by the scene and captured by a depth sensing camera (). The light pattern captured by the depth sensing camera is differentfrom the original known pattern because of the reflection by the scene,that is, the light pattern captured by the imaging camera is “deformed.”Based on the differences between the deformed pattern and the knownoriginal pattern, the depth sensing system can calculate the depthinformation of the scene. However, the light pattern being captured canbe further deformed due to factors other than the scene. For example, atemperature drift of a light source can cause a shift of the lightpattern. Such a shift of the light pattern tends to cause systematicbias of the calculated depth information of the scene.

SUMMARY

Introduced here are at least one apparatus and at least one method(collectively and individually, “the technique introduced here”) forcompensating for temperature shifts of a structured light pattern of adepth imaging system. In some embodiments, a depth imaging deviceincludes a light source, an imaging sensor and a processor. The lightsource emits light corresponding to a pattern. A temperature drift ofthe light source can cause a shift of the illumination dot pattern. Theimaging sensor receives the light reflected by environment in front ofthe depth imaging device and generates a depth map including a pluralityof pixel values corresponding to depths of the environment relative tothe depth imaging device. The processor estimates the shift of thepattern based on a polynomial model depending on the temperature driftof the light source. The processor further adjusts the depth map basedon the shift of the pattern.

In certain embodiments, the polynomial model can be a global model thatincludes a first polynomial to predict shifts in a first direction and asecond polynomial to predict shifts in a second direction. Each of thefirst and second polynomials includes a plurality of cubic terms. Eachof the cubic terms includes a product of multiplying three variablesthat include a temperature drift from a reference temperature, ax-coordinate in a reference image corresponding to the pattern, or ay-coordinate in the reference image.

In certain embodiments, the polynomial model can be an individualregression model. The individual regression model includes a pluralityof polynomial sets. Each of the polynomial sets predicts a shift of oneof the objects of the pattern due to temperature drift. Each of thepolynomial sets including a polynomial to estimate a shift along a firstdirection and another polynomial to estimate a shift along a seconddirection.

Other aspects of the disclosed embodiments will be apparent from theaccompanying figures and detailed description.

This Summary is provided to introduce a selection of concepts in asimplified form that are further explained below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present disclosure are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements.

FIG. 1 shows an example of an environment in which a virtual reality(VR) or augmented reality (AR) enabled head-mounted display device(hereinafter “HMD device”) can be used.

FIG. 2 illustrates a perspective view of an example of an HMD device.

FIG. 3 shows a portion of a sensor assembly of an HMD device including adepth camera and an illumination module.

FIG. 4 shows a speckle pattern as an example of a structured lightpattern that can be produced in a structured light process.

FIG. 5 illustrates shifts in diffraction angle shift with temperaturefor a structured light illuminator.

FIG. 6 illustrates an example of a process of compensating temperatureshifts of a structured light pattern for a depth imaging system.

FIG. 7 illustrates an example of a processor for generating compensatedshifts of a structured light pattern.

FIG. 8 shows a high-level example of a hardware architecture of a systemthat can be used to implement any one or more of the functionalcomponents described herein.

DETAILED DESCRIPTION

In this description, references to “an embodiment,” “one embodiment” orthe like mean that the particular feature, function, structure orcharacteristic being described is included in at least one embodimentintroduced here. Occurrences of such phrases in this specification donot necessarily all refer to the same embodiment. On the other hand, theembodiments referred to also are not necessarily mutually exclusive.

The following description generally assumes that a “user” of a displaydevice is a human. Note, however, that a display device of the disclosedembodiments can potentially be used by a user that is not human, such asa machine or an animal. Hence, the term “user” can refer to any of thosepossibilities, except as may be otherwise stated or evident from thecontext. Further, the term “optical receptor” is used here as a generalterm to refer to a human eye, an animal eye, or a machine-implementedoptical sensor designed to detect an image in a manner analogous to ahuman eye.

Virtual reality (VR) or augmented reality (AR) enabled head-mounteddisplay (HMD) devices and other near-to-eye display systems may includetransparent display elements that enable users to see concurrently boththe real world around them and AR content displayed by the HMD devices.An HMD device may include components such as light-emission elements(e.g., light emitting diodes (LEDs)), waveguides, various types ofsensors, and processing electronics. HMD devices may further include oneor more imager devices to generate images (e.g., stereo pair images for3D vision) in accordance with the environment of a user wearing the HMDdevice, based on measurements and calculations determined from thecomponents included in the HMD device.

An HMD device may also include a depth imaging system (also referred toas depth sensing system or depth imaging device) that resolves distancebetween the HMD device worn by a user and physical surfaces of objectsin the user's immediate vicinity (e.g., walls, furniture, people andother objects). The depth imaging system may include a structured lightor ToF camera that is used to produce a 3D image of the scene. Thecaptured image has pixel values corresponding to the distance betweenthe HMD device and points of the scene.

The HMD device may include an imaging device that generates holographiccontent based on the scanned 3d scene, and that can resolve distances,for example, so that holographic objects appear at specific locationsrelative to physical objects in the user's environment. 3D imagingsystems can also be used for object segmentation, gesture recognition,and spatial mapping. The HMD device may also have one or more displaydevices to display the generated images overlaid on the field of view ofan optical receptor of a user when the HMD device is worn by the user.Specifically, one or more transparent waveguides of the HMD device canbe arranged so that they are positioned to be located directly in frontof each eye of the user when the HMD device is worn by the user, to emitlight representing the generated images into the eyes of the user. Withsuch a configuration, images generated by the HMD device can be overlaidon the user's three-dimensional view of the real world.

FIGS. 1 through 8 and related text describe certain embodiments of anillumination module in the context of near-to-eye display systems.However, the disclosed embodiments are not limited to near-to-eyedisplay systems and have a variety of possible applications, includingany active illumination systems (i.e., actively using light sources)such as used in active light projection systems or any active cameramodules. All such applications, improvements, or modifications areconsidered within the scope of the concepts disclosed here.

FIG. 1 schematically shows an example of an environment in which an HMDdevice can be used. In the illustrated example, the HMD device 10 isconfigured to communicate data to and from an external processing system12 through a connection 14, which can be a wired connection, a wirelessconnection, or a combination thereof. In other use cases, however, theHMD device 10 may operate as a standalone device. The connection 14 canbe configured to carry any kind of data, such as image data (e.g., stillimages and/or full-motion video, including 2D and 3D images), audio,multimedia, voice, and/or any other type(s) of data. The processingsystem 12 may be, for example, a game console, personal computer, tabletcomputer, smartphone, or other type of processing device. The connection14 can be, for example, a universal serial bus (USB) connection, Wi-Ficonnection, Bluetooth or Bluetooth Low Energy (BLE) connection, Ethernetconnection, cable connection, digital subscriber line (DSL) connection,cellular connection (e.g., 3G, LTE/4G or 5G), or the like, or acombination thereof. Additionally, the processing system 12 maycommunicate with one or more other processing systems 16 via a network18, which may be or include, for example, a local area network (LAN), awide area network (WAN), an intranet, a metropolitan area network (MAN),the global Internet, or combinations thereof.

FIG. 2 shows a perspective view of an HMD device 20 that can incorporatethe features being introduced here, according to certain embodiments.The HMD device 20 can be an embodiment of the HMD device 10 of FIG. 1.The HMD device 20 has a protective sealed visor assembly 22 (hereafterthe “visor assembly 22”) that includes a chassis 24. The chassis 24 isthe structural component by which display elements, optics, sensors andelectronics are coupled to the rest of the HMD device 20. The chassis 24can be formed of molded plastic, lightweight metal alloy, or polymer,for example.

The visor assembly 22 includes left and right AR displays 26-1 and 26-2,respectively. The AR displays 26-1 and 26-2 are configured to displayimages overlaid on the user's view of the real-world environment, forexample, by projecting light into the user's eyes. Left and right sidearms 28-1 and 28-2, respectively, are structures that attach to thechassis 24 at the left and right open ends of the chassis 24,respectively, via flexible or rigid fastening mechanisms (including oneor more clamps, hinges, etc.). The HMD device 20 includes an adjustableheadband (or other type of head fitting) 30, attached to the side arms28-1 and 28-2, by which the HMD device 20 can be worn on the user'shead.

The chassis 24 may include various fixtures (e.g., screw holes, raisedflat surfaces, etc.) to which a sensor assembly 32 and other componentscan be attached. In some embodiments the sensor assembly 32 is containedwithin the visor assembly 22 and mounted to an interior surface of thechassis 24 via a lightweight metal frame (not shown). A circuit board(not shown in FIG. 2) bearing electronics components of the HMD 20(e.g., microprocessor, memory) can also be mounted to the chassis 24within the visor assembly 22.

The sensor assembly 32 includes a depth camera 34 and an illuminationmodule 36 of a depth imaging system. The illumination module 36 emitslight to illuminate a scene. Some of the light reflects off surfaces ofobjects in the scene, and returns back to the imaging camera 34. In someembodiments such as an active stereo system, the assembly can includetwo or more cameras. The depth camera 34 captures the reflected lightthat includes at least a portion of the light from the illuminationmodule 36.

The “light” emitted from the illumination module 36 is electromagneticradiation suitable for depth sensing and should not directly interferewith the user's view of the real world. As such, the light emitted fromthe illumination module 36 is typically not part of the human-visiblespectrum. Examples of the emitted light include infrared (IR) light tomake the illumination unobtrusive. Sources of the light emitted by theillumination module 36 may include LEDs such as super-luminescent LEDs,laser diodes, or any other semiconductor-based light source withsufficient power output.

The depth camera 34 may be or include any image sensor configured tocapture light emitted by an illumination module 36. The depth camera 34may include a lens that gathers reflected light and images theenvironment onto the image sensor. An optical bandpass filter may beused to pass only the light with the same wavelength as the lightemitted by the illumination module 36. For example, in a structuredlight depth imaging system, each pixel of the depth camera 34 may usetriangulation to determine the distance to objects in the scene. Any ofvarious approaches known to persons skilled in the art can be used fordetermining the corresponding depth calculations.

The HMD device 20 includes electronics circuitry (not shown in FIG. 2)to control the operations of the depth camera 34 and the illuminationmodule 36, and to perform associated data processing functions. Thecircuitry may include, for example, one or more processors and one ormore memories. As a result, the HMD device 20 can provide surfacereconstruction to model the user's environment, or be used as a sensorto receive human interaction information. With such a configuration,images generated by the HMD device 20 can be properly overlaid on theuser's 3D view of the real world to provide a so-called augmentedreality. Note that in other embodiments the aforementioned componentsmay be located in different locations on the HMD device 20.Additionally, some embodiments may omit some of the aforementionedcomponents and/or may include additional components not discussed abovenor shown in FIG. 2. In some alternative embodiments, the aforementioneddepth imaging system can be included in devices that are not HMDdevices. For example, depth imaging systems can be used in motionsensing input devices for computers or game consoles, automotive sensingdevices, earth topography detectors, robots, etc.

The HMD device 20 can also include a temperature sensor 33. Thetemperature sensor can detect a temperature drift of, e.g., the depthcamera 23, the illumination module 36, another optical component of theHMD device 20, or an environment of the HMD device 20.

FIG. 3 shows a portion of the sensor assembly 32 of the HMD device 20.In particular, the sensor assembly 32 includes sensors and electronicsmounted to a circuit board, which can be mounted to the chassis 24 asmentioned above. The sensors mounted to the circuit board include thedepth camera 34 and the illumination module 36. Other sensors that maybe included in the sensor assembly 32 but are not shown in the figuresor discussed further may include head-tracking cameras, visible spectrumcameras, ambient light sensors, and the like. Some or all of these othersensors may also be mounted to the sensor assembly 32.

The locations and positions of the illumination module 36 and the depthcamera 34 relative to each other as shown in FIG. 3 are merely examplesof a configuration used for depth sensing; other configurations arepossible in the context of the technique introduced here.

Shift of Structured Light Pattern Due to Temperature Variation

A depth imaging system can include a light source for providingstructured light for projecting a known pattern of light onto a scene.For example, a laser illumination source, as illustrated in FIG. 3, canoperate as the light source for providing the structured light. FIG. 4shows a dot pattern as an example of a structured light pattern that canbe produced in a structured light depth evaluation. The speckle pattern400 (also referred to as dot pattern or pattern of dots) can begenerated by, e.g., diffractive optical elements (DOEs). When theenvironment reflects the light, the pattern illuminated by thestructured light source is deformed before the pattern being captured bythe depth camera. The depth imaging system can calculate the depths ofsurfaces and the environment, relative to the imaging system, based onthe displacements of the dots in the pattern 400, from the originalknown speckle pattern 400 to a deformed pattern captured by the depthcamera.

However, the speckle pattern may also drift due to temperature variationof the laser illuminator. In particular, the wavelength of the lightsource (e.g. laser diode) used for illumination may be sensitive tochanges in the semiconductor junction temperature. For example, if anillumination power of the light source changes, the junction temperaturewill shift. In turn, the wavelength of the light source will also shift.For example, a wavelength shift for a 850 nm laser diode due to atemperature drift is approximately 0.3 nm/° C.

The shift of the light wavelength causes shifting of dots in the specklepattern. For example, for a speckle pattern generated by a holographicdiffraction grating, the diffraction angle is

${{\theta\; m} = {\arcsin\left( \frac{m\;\lambda}{d} \right)}};$where λ is the wavelength of the illumination, and d is the gratingpitch, m is the diffraction order of the light diffracted from thediffraction grating. As the function of the diffraction angle shows,light of higher diffraction orders experience greater shifts with thelaser wavelength, which is influenced by the device temperature. FIG. 5illustrates shifts in diffraction angles for a sample structured lightsystem. The sample structured light system includes, e.g., a structurelight source with an illumination wavelength of 850 nm and a diffractiongrating with a grating dimension of 120 um.

Such drift of the speckle pattern due to temperature variation,especially if occurring in the horizontal direction (relative to thecamera orientation), tends to cause systematic bias of the captureddepth information. Furthermore, significant drift in the verticaldirection can cause conventional scanline searching algorithms forgenerating depth maps to fail.

Some conventional systems try to avoid temperature variation bystabilizing the laser diode temperature by using thermoelectric coolers(TECs). However, the additional size and power consumption of the TECsmakes such a solution unacceptable for many applications, such as mobileelectronic devices (e.g., HMDs).

According to the technique introduced here, therefore, a structuredlight depth imaging system can use a temperature compensation model toenable the system to perform accurately in different ambient temperatureenvironments, without the need for expensive per-device calibration orbulky and power-hungry TECs. For applying the temperature compensationmodel, the system collects dot pattern images for different ambienttemperature, applies a regression model for each dot position, andrecovers a reference view of the pattern for any given ambienttemperature before applying stereo searching algorithm for calculatingdepth information. The temperature compensation model can be, e.g., apolynomial model. Two examples of the temperature compensationpolynomial model are disclosed here: a global model and an individualregression model.

Although various embodiments discloses a structured light sourceprojecting a speckle pattern including a plurality of dots as anexample, any of various known or convenient patterns could be used asthe pattern for structured light. For example, the technology disclosedhere can be applied to a pattern including objects with differentshapes.

Global Model

The structured light depth imaging system can apply a global model toestimate shift of structured light pattern due to temperature variation(such a shift is also referred to as temperature shift). The globalmodel uses a set of polynomials for modelling temperature shifts of allfeatures in the structured light pattern. For example, the global modelcan use two polynomials to model, respectively, horizontal shift andvertical shifts of dots in the structured light pattern. Each polynomialdepends on, e.g., coordinates of the individual dot in the originalknown pattern (also referred to as original reference image) and atemperature drift between a current temperature and a referencetemperature. The global model may omit effects of local device variationand geometric lens distortion, but is still efficient enough for afurther line prediction process to calculate the depth information.

In one embodiment, the polynomial for modeling horizontal shift of theindividual dot can be:x′=a ₁ t ³ +a ₂ tx ² +a ₃ ty ² +a ₄ txy+a ₅ t ² x+a ₆ t ² y+a ₇ tx+a ₈ty+a ₉ t ² +a ₁₀ t+x.  (1).

Similarly, the polynomial for modeling vertical shift of the individualdot can be:y′=b ₁ t ³ +b ₂ tx ² +b ₃ ty ² +b ₄ txy+b ₅ t ² x+b ₆ t ² y+b ₇ tx+b ₈ty+b ₉ t ² +b ₁₀ t+y.  (2).Coordinates x and y are the positional coordinates of the dots in thedot pattern in the original reference image; x′ and y′ are thepositional coordinates of the dots in the dot pattern after thetemperature drift (also referred to as temperature-shifted coordinates);and t is the temperature drift between a current temperature and areference temperature. The current temperature can be measured by, e.g.,a temperature sensor on the device. The weight coefficients for theweighted linear combination (a_(l),b_(l) l=1 . . . 10) are modelparameters (also referred to as weight coefficients). The polynomials(1) and (2) of the global model include cubic terms (e.g.,a₁t³,a₂tx²,a₃ty²,a₄txy,a₅t²x,a₆t²y) to model the interaction betweencoordinates x and y and temperature drift t. Each cubic term includes aproduct of multiplying three variables; the variables can include, e.g.,coordinate x, coordinate y or temperature drift t.

A training algorithm, e.g., a linear regression algorithm (also referredto as a linear regression process), can be used to estimate the modelparameters (a_(l),b_(l) l=1 . . . 10), by feeding known observations (x,y, t) and known response variables (x′, y′) and minimizing a differencebetween the calculate temperature-shifted coordinates and the fed knownresponse coordinates. In other words, the cost function to be minimizedduring training is indicative of the difference between the calculatetemperature-shifted coordinates and the known response coordinates.

Alternatively, a training algorithm can minimize the difference betweena known depth map and a calculated depth map. For example, to determinethe model parameters (a_(l),b_(l) l=1 . . . 10), the polynomials (1) and(2) of the global model can be trained using the dot pattern in theoriginal reference images, and shifted dot patterns in calibrationreference images with different temperature drifts. The shifted dotpatterns in calibration reference images are reflected by a calibrationenvironment. The depth map of the calibration environment is known priorto the training. (For instance, the calibration environment can be aflat surface). Because the depth map is known, the model can adjust themodel parameters during training so that the model predicts a shift ofdot pattern due to temperature variation that leads to the known depthmap. In other words, the model parameters are optimized during trainingto minimize a cost function. The cost function is indicative of adifference between the known depth map and a calculated depth map. Thecalculated depth map is calculated based on a temperature-shift of dotpattern due to temperature variation predicted by the model.

In some embodiments, the polynomials of the model can include termsusing other variables. For example, some polynomial terms can include anangle between the light ray of each dot and an optical center of thedepth imaging system. The model also can user a polar coordinate systemto replace the Cartesian coordinate system.

In some embodiments, the polynomials of the model can include terms oforders higher than three in polynomials (1) and (2). For example, somepolynomial terms can be quartic terms (order of 4), which includeproduct of multiplying four variables. The terms of higher orders canmodel more complex interactions between x, y and t.

In some embodiments, the polynomials of the model can just include termsof orders less than 3. The polynomials with quadratic terms (order of 2)use less computing power to calculate and are suitable for situationswhere the interactions between x, y and t are not complex. Thepolynomials with quadratic terms can be, e.g.: x′=a₁tx+a₂ty+a₃t²+a₄t+x;y′=b₁tx+b₂ty+b₃t²+b₄t+y.

Individual Regression Model

The global model does not capture the variation of shifts for eachindividual dot, and also does not model dots that are local outliers andthat do not follow the global model. Alternatively, therefore, anindividual regression model can include polynomials for modelingtemperature shifts for each individual dot:x _(i) ′=a _(1i) t ² +b _(1i) xt+c _(1i) yt+d _(1i) t+x;  (3);y _(i) ′=a _(2i) t ² +b _(2i) xt+c _(2i) yt+d _(2i) t+y;  (4);wherein i is the index number of an individual dot among the dots of thepattern.

The polynomials (3) and (4) of the individual regression model can usequadratic terms that use less computing power than terms of higherorders. As a result, the polynomials (3) and (4) of the individualregression model can include a total of 10n model parameters (where n isthe number of dots). In contrast, the global model can include fewermodel parameters. For example, one embodiment of the global model usingthe polynomials (1) and (2) includes 20 model parameters.

In some embodiments, to reduce the memory usage due to the 10n modelparameters, the individual regression model can exploit the spatialconsistency between dots in proximity and subsample the parameter space.In other words, the individual regression model can include onlypolynomials for modeling temperature shifts for a subset of the selecteddots in the structured light pattern. For the remaining dots, thetemperature shifts can be predicted using polynomials of a neighborselected dot, because generally dots in proximity have consistenttemperature shifts.

FIG. 6 illustrates an example of a process 600 of compensatingtemperature shifts of a structured light pattern for a depth imagingsystem. In step 605 of the process 600, a structured light source of thedepth imaging system emits light corresponding to a known pattern. Theknown pattern can include a plurality of objects (e.g. dots). In step610, an imaging sensor receives the light reflected by environment inthe field of view of the of the depth imaging device. In step 615, theimaging sensor generates an image based on the received light a depthmap. The depth map includes a plurality of pixel values corresponding todepths of the environment relative to the depth imaging device. In step620, a temperature sensor of the depth imaging system detects that atemperature of the structured light source has drifted from a referencetemperature. In step 625, in response to the temperature drift, aprocessor predicts the temperature shifts of the objects of the patternbased on a polynomial model. Such a polynomial model can be, e.g., theglobal model or the individual regression model disclosed here. In step630, the processor adjusts the depth map based on the temperature shiftsof the objects of the patterns.

In some embodiments, the system can adjust the depth map by combiningthe global model and the individual regression model. For example, inone embodiment, the system can use the individual regression model topredict temperature shifts of a subset of key dots of the pattern, anduse the global model to predict temperature shifts of the remainingdots. Alternatively, the system can use both the global model and theindividual regression model to predict two sets of temperature shifts ofthe dots. The system can then determine the temperature shifts using thetwo sets of predicted results. For example, the system may determine thetemperature shifts of the dots by averaging the values of the two setsof temperature shifts.

The system can detect a distorted depth map if both the conditions ofstep 530 and step 960 are satisfied. In another embodiment, the systemcan detect a distorted depth map if either of the conditions of step 530and step 960 is satisfied.

In some embodiments, the global model or the individual regression modelcan use functions other than polynomials. For example, the functions caninclude terms with exponents that are not positive integers.

Temperature Interpolation Model

Alternatively, or additionally, the temperature-shifted coordinates canbe estimated based on interpolation. The temperature interpolation modelcan set up a plurality of key temperatures T=[t₁ . . . , t_(m)], where mis the number of key temperatures. For each key temperature T, a set oftemperature-shifted coordinates (x′, y′) or (x_(i)′, y_(i)′) environmentare recorded. For a temperature t that is not a key temperature, themodel identifies two nearest neighbor key temperatures that arerespectively above and below the temperature t. The temperature-shiftedcoordinates for temperature t are the linear interpolations between thecorresponding temperature-shifted coordinates for the two nearestneighbor key temperatures. In some other embodiments, splines orpolynomials can be used for interpolation, instead of a linearinterpolation.

In some alternative embodiments, for temperature-shifts that are notsmooth enough to be described by the disclosed models, a per-dot lookuptable can be generated. For each dot with coordinates x, y andtemperature t, the lookup table stores the temperature-shiftedcoordinates x′, y′.

FIG. 7 illustrates an example of a processor for generating compensatedshifts of a structured light pattern. To predict the temperature shift,the processor 700 of a depth sensing system receives the polynomials asinputs. The polynomials can be, e.g., the polynomials (1) and (2) of theglobal model or the polynomials (3) and (4) of the individual regressionmodel. The received inputs include the trained model parameters. Theprocessor 700 further receives the temperature drift 715 and theoriginal pattern being projected 710 as inputs. The temperature drift715 can a drift of temperature of the light source of the system, adrift of temperature of an optical component of the system, or a driftof temperature of an environment where the system is located. Theinformation of the original pattern 710 can include positionalcoordinates of features on the pattern 710. The processor 700 uses theinputs to predict a temperature-shifted pattern 720. The information ofthe temperature-shifted pattern 720 can include shifted positionalcoordinates of the pattern features. Furthermore, using thetemperature-shifted pattern 720, the processor 700 can adjust the depthmap using a process illustrated in FIG. 6.

FIG. 8 shows a high-level example of a hardware architecture of aprocessing system that can be used to implement to perform the disclosedfunctions (e.g., steps of the process 600). The processing systemillustrated in FIG. 8 can be part of an NED device. One or multipleinstances of an architecture such as shown in FIG. 8 (e.g., multiplecomputers) can be used to implement the techniques described herein,where multiple such instances can be coupled to each other via one ormore networks.

The NED device 800 includes a depth camera 823 and an illuminationmodule 836, similar to the HMD device 20 as illustrated in FIG. 2. TheNED device 800 can also include a temperature sensor 833. Thetemperature sensor 833 can detect a temperature drift of, e.g., a depthcamera, a light source, an optical component of the NED device 800, oran environment of the NED device 800.

The illustrated processing system 800 includes one or more processors810, one or more memories 811, one or more communication device(s) 812,one or more input/output (I/O) devices 813, and one or more mass storagedevices 814, all coupled to each other through an interconnect 815. Theinterconnect 815 may be or include one or more conductive traces, buses,point-to-point connections, controllers, adapters and/or otherconventional connection devices. Each processor 810 controls, at leastin part, the overall operation of the processing device 800 and can beor include, for example, one or more general-purpose programmablemicroprocessors, digital signal processors (DSPs), mobile applicationprocessors, microcontrollers, application specific integrated circuits(ASICs), programmable gate arrays (PGAs), or the like, or a combinationof such devices.

Each memory 811 can be or include one or more physical storage devices,which may be in the form of random access memory (RAM), read-only memory(ROM) (which may be erasable and programmable), flash memory, miniaturehard disk drive, or other suitable type of storage device, or acombination of such devices. Each mass storage device 814 can be orinclude one or more hard drives, digital versatile disks (DVDs), flashmemories, or the like. Each memory 811 and/or mass storage 814 can store(individually or collectively) data and instructions that configure theprocessor(s) 810 to execute operations to implement the techniquesdescribed above. Each communication device 812 may be or include, forexample, an Ethernet adapter, cable modem, Wi-Fi adapter, cellulartransceiver, baseband processor, Bluetooth or Bluetooth Low Energy (BLE)transceiver, or the like, or a combination thereof. Depending on thespecific nature and purpose of the processing system 800, each I/Odevice 813 can be or include a device such as a display (which may be atouch screen display), audio speaker, keyboard, mouse or other pointingdevice, microphone, camera, etc. Note, however, that such I/O devicesmay be unnecessary if the processing device 800 is embodied solely as aserver computer.

In the case of a user device, a communication device 812 can be orinclude, for example, a cellular telecommunications transceiver (e.g.,3G, LTE/4G, 5G), Wi-Fi transceiver, baseband processor, Bluetooth or BLEtransceiver, or the like, or a combination thereof. In the case of aserver, a communication device 812 can be or include, for example, anyof the aforementioned types of communication devices, a wired Ethernetadapter, cable modem, DSL modem, or the like, or a combination of suchdevices.

The machine-implemented operations described above can be implemented atleast partially by programmable circuitry programmed/configured bysoftware and/or firmware, or entirely by special-purpose circuitry, orby a combination of such forms. Such special-purpose circuitry (if any)can be in the form of, for example, one or more application-specificintegrated circuits (ASICs), programmable logic devices (PLDs),field-programmable gate arrays (FPGAs), system-on-a-chip systems (SOCs),etc.

Software or firmware to implement the embodiments introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium,” as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

EXAMPLES OF CERTAIN EMBODIMENTS

Certain embodiments of the technology introduced herein are summarizedin the following numbered examples:

1. A depth imaging device including: a light source to emit lightcorresponding to a predetermined pattern; an imaging sensor to receivethe light reflected by the environment of the depth imaging device andthat, when in operation, generates a depth map including a plurality ofpixel values corresponding to depths of the environment relative to theimaging device; and a processor that, when in operation, estimates ashift of the pattern by using a polynomial model depending on thetemperature drift of the light source and adjusts the depth map based onthe shift of the pattern.

2. The device of example 1, wherein a temperature drift of the lightsource, an optical component of the depth imaging device, or anenvironment of the depth imaging device causes a shift of the pattern.

3. The device of example 1 or 2, further including: a temperature sensorto measure a temperature of the light source.

4. The device in any of the preceding examples 1 through 3, wherein thelight source is a structured light source to emit light corresponding toa known pattern.

5. The device in any of the preceding examples 1 through 4, wherein thepattern is a speckle pattern corresponding to a reference imageincluding a plurality of dots, each of the dots having known coordinatesin the reference image.

6. The device in any of the preceding examples 1 through 5, wherein thepolynomial model is used to compensate for shifts in the pattern forgenerating an accurate depth map.

7. The device in any of the preceding examples 1 through 6, where thepolynomial model includes a first polynomial and a second polynomial,the first and second polynomials being functions of a plurality ofvariables, the variables including a temperature drift from a referencetemperature and positional coordinates in the reference image.

8. The device of example 7, wherein the pattern includes a plurality ofobjects and the polynomial model includes the first and secondpolynomials to estimate shifts of the objects of the pattern due to thetemperature drift.

9. The device of example 7 or 8, wherein each of the first and secondpolynomials includes a plurality of cubic terms to model interactionsbetween the positional coordinates in the reference image and thetemperature drift, each of the cubic terms including a product ofmultiplying three variables.

10. The device in any of the preceding examples 7 through 9, whereineach of the first and second polynomials is a weighted combination ofterms depending on the temperature drift, a first positional coordinatein the reference image, or a second positional coordinate in thereference image; and wherein the processor trains weight coefficients ofthe first and second polynomials using a training data set, the trainingdata set including values of the positional coordinates in the referenceimage, values of the temperature drifts, and corresponding known valuesof shifted positional coordinates in shifted images that correspond toshifts of the pattern due to the temperature drift.

11. The device in any of the preceding examples 1 through 10, whereinthe pattern includes a plurality of objects; and wherein the polynomialmodel includes a plurality of function sets, each of the function setsis predictive of shifts of one of the objects of the pattern due to thetemperature drift, each of the function sets including a functionpredictive of a shift along a first direction and another functionpredictive of a shift along a second direction.

12. The device of example 11, wherein each function of the function setsincludes a plurality of quadratic terms to model interactions betweenthe positional coordinates in the reference image and the temperaturedrift, each of the quadratic terms including a product of multiplyingtwo variables.

13. The device of example 11 or 12, wherein each function of thefunction sets is a weighted combination of terms depending on thetemperature drift, a first coordinate in the reference image, or asecond coordinate in the reference image; and wherein weightcoefficients of each function are trained based on a plurality of knownshifts of the pattern due to the temperature drift.

14. The device in any of the preceding examples 11 through 13, whereinthe plurality of function sets are predictive of shifts of a subset ofsampling objects within the pattern; and wherein the polynomial modelestimates a shift of a remaining object using one function set thatcorresponds to a sampling object within the subset, the sampling objectin proximity to the remaining object in the reference image.

15. An apparatus for compensating for temperature shifts of a structuredlight pattern, including: means for emitting light corresponding to aknown pattern by a structured light source of a depth imaging device,the known pattern including a plurality of objects that have shifts inthe pattern due to a temperature drift of the structured light source;means for receiving the light reflected by an environment of the depthimaging device; means for generating, based on the light, a depth mapincluding a plurality of pixel values corresponding to depths of objectsin the environment relative to the depth imaging device; means forpredicting the temperature shifts of the objects of the pattern based ona polynomial model; and means for adjusting the depth map based on thetemperature shifts of the objects of the patterns.

16. The apparatus of example 15, further including: means foridentifying a location of an object in the environment by using thedepth map that has been adjusted.

17. The apparatus of example 15 or 16, further including: means fortraining the polynomial model by optimizing model parameters of thepolynomial model using a training data set, the training data setincluding known shifts of the objects due to temperature drifts andcorresponding temperature drifts from a reference temperature.

18. The apparatus in any of the preceding examples 15 through 17,wherein the polynomial model includes a first polynomial predictive ofshifts in a first direction and a second polynomial predictive of shiftsin a second direction, each of the first and second polynomialsincluding a plurality of cubic terms, each of the cubic terms includinga product of multiplying three variables that include a temperaturedrift from a reference temperature, a x coordinate in a reference imagecorresponding to the pattern, or a y coordinate in the reference image;and wherein the means for predicting includes: means for predicting thetemperature shifts of the objects of the pattern by applying the firstand second polynomials to each of the objects of the pattern.

19. The apparatus in any of the preceding examples 15 through 18,wherein the polynomial model includes a plurality of polynomial sets,each of the polynomial sets predictive of shifts of one of the objectsof the pattern due to temperature drift, each of the polynomial setsincluding a polynomial to estimate a shift along a first direction andanother polynomial to estimate a shift along a second direction; andwherein the means for predicting includes: means for predicting thetemperature shifts of the objects of the pattern by applying eachpolynomial set of the plurality of polynomial sets to a correspondingobject of the objects of the pattern.

20. A head-mounted display device including: a display that, when inoperation, outputs an image to an eye of a user; a light source to emitlight corresponding to a pattern including a plurality of features,wherein a temperature drift of the light source, an optical component ofthe head-mounted display device, or an environment of the head-mounteddisplay device causes a shift of the pattern; an imaging sensor toreceive the light reflected by the environment of the head-mounteddisplay device and that, when in operation, generates a depth mapincluding a plurality of pixel values corresponding to depths of theenvironment relative to the head-mounted display device; and a processorthat, when in operation, estimates the shifts of the features of thepattern by using a polynomial model depending on the temperature driftof the light source and adjusts the depth map based on the shifts of thefeatures.

Any or all of the features and functions described above can be combinedwith each other, except to the extent it may be otherwise stated aboveor to the extent that any such embodiments may be incompatible by virtueof their function or structure, as will be apparent to persons ofordinary skill in the art. Unless contrary to physical possibility, itis envisioned that (i) the methods/steps described herein may beperformed in any sequence and/or in any combination, and that (ii) thecomponents of respective embodiments may be combined in any manner.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims, and other equivalent features and acts are intended to be withinthe scope of the claims.

What is claimed is:
 1. A depth imaging device, comprising: an imagingsensor configured to: receive light reflected by an environment within afield of view of the depth imaging device; and generate, based on thelight, a depth map including a plurality of pixel values correspondingto depths of the environment relative to the depth imaging device; atemperature sensor configured to measure a temperature driftcorresponding to the light; and a processor configured to: estimate ashift of the light based on the temperature drift using one or moretemperature compensation models; and adjust the depth map based on theshift of the light.
 2. The depth imaging device of claim 1, wherein theprocessor is further configured to: estimate a shift of a pattern of thelight based on the temperature drift; and adjust the depth map furtherbased on the shift of the pattern, wherein the pattern is a specklepattern corresponding to a reference image including a plurality ofdots, each dot of the plurality of dots having known coordinates in thereference image.
 3. The depth imaging device of claim 1, wherein theprocessor is further configured to: estimate a shift of a pattern of thelight based on the temperature drift; and adjust the depth map furtherbased on the shift of the pattern, wherein the shift of the pattern isestimated by using a polynomial model depending on the temperaturedrift.
 4. The depth imaging device of claim 3, wherein the polynomialmodel is used to compensate for shifts in the pattern for generating anaccurate depth map.
 5. The depth imaging device of claim 3, wherein thepattern is a light pattern corresponding to a reference image includinga plurality of objects; and wherein the polynomial model is used toestimate shifts of one or more objects of the plurality of objects dueto the temperature drift.
 6. The depth imaging device of claim 3,wherein the pattern is a light pattern corresponding to a referenceimage; and wherein the polynomial model includes polynomials that arefunctions of a plurality of variables, wherein the plurality ofvariables include the temperature drift from a reference temperature andpositional coordinates in the reference image.
 7. The depth imagingdevice of claim 6, wherein the polynomial model includes one or morepolynomials having a plurality of cubic terms to model interactionsbetween the positional coordinates in the reference image and thetemperature drift, each of the plurality of cubic terms including aproduct of multiplying three variables of the plurality of variables. 8.The depth imaging device of claim 6, wherein the polynomial modelincludes one or more polynomials having a weighted combination of termsdepending on the temperature drift; and wherein the processor is furtherconfigured to train weight coefficients of the one or more polynomialsusing a training data set, the training data set including values of thepositional coordinates in the reference image, values of the pluralityof variables of the one or more polynomials associated with thetemperature drift, and corresponding known values of shifted positionalcoordinates in shifted images that correspond to shifts of the patterndue to the temperature drift.
 9. The depth imaging device of claim 1,wherein the processor is further configured to: estimate a shift of apattern of the light based on the temperature drift; and adjust thedepth map further based on the shift of the pattern, wherein the patternis a light pattern corresponding to a reference image including aplurality of objects; and wherein the shift of the pattern is estimatedby using a polynomial model, wherein the polynomial model includes afunction set, each function of the function set being predictive ofshifts of a corresponding object of the plurality of objects due to thetemperature drift, the function set including a first functionpredictive of a shift along a first direction and a second functionpredictive of a shift along a second direction.
 10. The depth imagingdevice of claim 9, wherein each function of the function set includes aplurality of quadratic terms to model interactions between positionalcoordinates of the reference image and the temperature drift, eachquadratic term of the plurality of quadratic terms including a productof multiplying two variables.
 11. The depth imaging device of claim 9,wherein each function of the function set is a weighted combination ofterms depending on the temperature drift, a first coordinate in thereference image, or a second coordinate in the reference image; andwherein weight coefficients of each function are trained based on aplurality of known shifts of the pattern due to the temperature drift.12. The depth imaging device of claim 9, wherein the function set ispredictive of shifts of a subset of sampling objects of the plurality ofobjects; and wherein the polynomial model estimates a shift of an objectof the plurality of objects using one function set that corresponds to asampling object within the subset, the sampling object in proximity to aremaining object in the reference image.
 13. A method for compensatingfor temperature shift of a structured light pattern of a depth imagingsystem, the method comprising: receiving light reflected by anenvironment within a field of view of the depth imaging system;generating, based on the light, a depth map including a plurality ofpixel values corresponding to depths of the environment relative to thedepth imaging system; determining, by a temperature sensor, atemperature drift corresponding to the light; estimating a shift of thelight based on the temperature drift using one or more temperaturecompensation models; and adjusting the depth map based on the shift ofthe light.
 14. The method of claim 13, further comprising: estimating ashift of a pattern of the light based on the temperature drift; andadjusting the depth map further based on the shift of the pattern,wherein the pattern is a speckle pattern corresponding to a referenceimage including a plurality of dots, each dot of the plurality of dotshaving known coordinates in the reference image.
 15. The method of claim13, further comprising: estimating a shift of a pattern of the lightbased on the temperature drift; and adjusting the depth map furtherbased on the shift of the pattern, wherein the shift of the pattern isestimated by using a polynomial model depending on the temperaturedrift.
 16. The method of claim 15, wherein the polynomial model is usedto compensate for shifts in the pattern for generating an accurate depthmap.
 17. The method of claim 15, wherein the pattern is a light patterncorresponding to a reference image including a plurality of objects; andwherein the polynomial model is used to estimate shifts of one or moreobjects of the plurality of objects due to the temperature drift. 18.The method of claim 15, wherein the pattern is a light patterncorresponding to a reference image; and wherein the polynomial modelincludes polynomials that are functions of a plurality of variables,wherein the plurality of variables include the temperature drift from areference temperature and positional coordinates in the reference image.19. The method of claim 18, wherein the polynomial model includes one ormore polynomials having a plurality of cubic terms to model interactionsbetween the positional coordinates in the reference image and thetemperature drift, each of the plurality of cubic terms including aproduct of multiplying three variables of the plurality of variables.20. A head-mounted display device comprising: an imaging sensorconfigured to: receive light reflected by an environment within a fieldof view of the head-mounted display device; and generate, based on thelight, a depth map including a plurality of pixel values correspondingto depths of the environment relative to the head-mounted displaydevice; a temperature sensor configured to measure a temperature driftcorresponding to the light; and a processor configured to: estimate ashift of the light based on the temperature drift using one or moretemperature compensation models; and adjust the depth map based on theshift of the light.